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INTRODUCTION 

One of the most versatile tools designed for use on the International Space Station (ISS) is the 
Special Purpose Dexterous Manipulator (SPDM) robot. Operators for this system are trained at 
NASA Johnson Space Center (JSC) using a robotic simulator, the Dexterous Manipulator 
Trainer (DMT), which performs most SPDM functions under normal static Earth gravitational 
forces. The SPDM is controlled from a standard Robotic Workstation. A key feature of the 
SPDM and DMT is the Force/Moment Accommodation (FMA) system, which limits the contact 
forces and moments acting on the robot components, on its payload - an Orbital Replaceable 
Unit (ORU), and on the receptacle for the ORU. The FMA system helps to automatically 
alleviate any binding of the ORU as it is inserted or withdrawn from a receptacle, but it is limited 
in its correction capability. A successful ORU insertion generally requires that the reference 
axes of the ORU and receptacle be aligned to within approximately 0.25 inch and 0.5 degree of 
nominal values. The only guides available for the operator to achieve these alignment tolerances 
are views from any available video cameras. No special registration markings are provided on 
the ORU or receptacle, so the operator must use their intrinsic features in the video display to 
perform the pre-insertion alignment task. Since optimum camera views may not be available, 
and dynamic orbital lighting conditions may limit viewing periods, long times are anticipated for 
performing some ORU insertion or extraction operations. This study explored the feasibility of 
using augmented reality (AR) to assist with SPDM operations. Geometric graphical symbols 
were overlaid on the end effector (EE) camera view to afford cues to assist the operator in 
attaining adequate pre-insertion ORU alignment. 

METHOD 

Participants 

Twelve participants were recruited from the pool of certified robotic operators available at JSC. 
These subjects ranged widely in their experience on robotic systems. Some were relatively new 
employees who had recently completed training working with one of the robotic systems used on 
ISS. Others included JSC robotic system instructors with experience training astronauts to use 
the Space Shuttle and/or ISS robotic ann. All participants were familiar with Robotic 
Workstation controls and procedures. 

Experimental Design 

The experiment was a classical “mixed” factors design having one factor (order of presentation 
of treatments) between subjects and one factor (treatment type - overlay or no overlay) repeated 
within subjects. Each subject perfonned a total of eight trials, which were divided into two sets 
of four trials each. One set was performed without overlays, and the other with AR graphical 
overlays. Half of the subjects experienced a set of four trials with overlays (OV) followed by a 
set of four trials without overlays (NO). The balance of the subjects experienced the same sets in 
reverse order. 

All trials involved maneuvering the ORU from a predetennined starting position toward the 
same receptacle. The same set of four different starting positions was used in the same order for 
the trial groups under the two treatments. Starting positions for the trials were selected to 
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produce the same minimum time for completion (as determined by mechanical characteristics of 
the DMT). The objective dependent measures for this study are related to the ORU 
position/orientation at the pre-insertion position above the destination receptacle. These four 
measures included (1) roll alignment error (degrees) about the X-axis, (2) composite pitch/yaw 
alignment error (degrees), (3) translation alignment error in the YZ plane (inches), and (4) 
elapsed time to trial completion (seconds). Coordinate conventions for the experiment are 
described below. In addition to the objective data collected, subjective evaluations of 
experimental treatments and methods were gathered by means of a written questionnaire 
provided to each subject following the completion of the trials. Subjects were encouraged to 
record any comments regarding the experiment on the questionnaire. 

Data analysis including analysis of variance (ANOVA) and post hoc testing of means was 
performed for all the significant dependent variables. Means comparisons were perfonned using 
contrasts in order to circumvent assumptions of homogeneity of variances among sample 
populations when necessary (Maxwell & Delaney, pp. 145-150). 

EXPERIMENTAL CONDITIONS 

Apparatus 

Experimental trials were conducted using a single arm of the DMT and the cupola mockup in the 
JSC Multiuse Remote Manipulator Facility. Subjects operated the DMT using the Robotic 
Workstation in the cupola mockup (Figure 1) under the local supervision of the experiment 
administrator. A trained operator at the DMT console set up initial conditions for each trial and 
continually monitored operations. 



Figure 1. Robotic Workstation in Cupola Mockup 
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The video signal from the DMT EE camera was routed through the video board in a personal 
computer (PC) on its way to the EE monitors in the cupola and at the DMT console. DMT 
telemetry data was also made available to the PC serial port from the DMT console. Custom 
software was developed to use the telemetry data to superimpose the overlays over the EE video 
signal presented to the monitors as required for the trial conditions. This software was developed 
using the open-source “ARToolKit” library (University of Washington Human Interface 
Technology Laboratory) to produce and manipulate the overlay graphics. 


Live video from end-iffeotor 
camera on DMT 



Composite video 
on USB line 


System Flowchart 


End-gffeelac telemetry 
(pitch, roll, yaw, X, Y, Z) 
from DMT. 
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hand 

controllers 
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DMT Augmented Reality System Flow Chart. 
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The ORU used for the experiment was the Remote Power Control Module (RPCM) mockup. 
Figure 2 shows the RPCM mockup above the open receptacle into which it was to be inserted. 
This payload was grappled by a Robotic Micro-conical Tool (RMCT), which was installed in the 
EE gripper of the DMT by the DMT console operator prior to the trials. Subjects were not 
required to grapple the ORU for this experiment. 



Figure 2. DMT Positioning RPCM Over its Receptacle 

Four cameras and a flood lamp were arrayed around the ORU receptacles, as shown in Figure 3. 
The flood lamp provided adverse, oblique lighting on the payload and the ORU receptacles to 
simulate stark, sunlit orbital lighting conditions. The cameras were mounted on pan/tilt units at 
different heights. The Robotic Workstation provided three monitors to accommodate views from 
any of these cameras and the EE camera, as shown in Figure 1 . 
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Figure 3. Camera and Flood Lamp Arrangement 
Coordinate Conventions 

Since the AR overlays appear only in the EE camera view (Figure 4), the operator can adopt an 
“internal” or “inside-out” (Huchingson, 1981) frame of reference and “fly” the grappled ORU 
using a right-handed set of orthogonal coordinates mapped to the EE camera monitor. 
Translations and rotations are referred to the monitor’s “vertical”, “horizontal”, and “in/out” 
axes. The in/out axis is called the X-axis, with the positive direction into the monitor screen. 

The horizontal axis is termed the Y-axis, with the positive direction toward the right of the 
screen. The remaining, vertical axis is called the Z-axis, which has its positive direction 
downward on the monitor. Rotations are referred about these orthogonal axes. A positive, right- 
handed rotation about the X-axis of the Robotic Workstation joystick, results in an ORU 
“rolling” motion to the right. A positive, “pitch-up” rotation of the joystick about the Y-axis, 
results in an upwardly tilting change in the EE monitor view. A positive “yaw-right” rotation of 
the joystick about the vertical Z-axis causes the EE monitor view to pan to the right. The origin 
for the axes is the “point of reference” (POR) for the EE/ORU, where the axis of the RMCT exits 
the “back” (connector) side of the ORU. The ORU rotational motions occur about the POR, and 
since the grapple target/camera axis is offset from the RMCT axis, there is a parallax 
consideration introduced. 

Description of the AR Overlays 

The system of AR symbol overlays described here was developed to speed the process of 
maneuvering a grappled ORU among bays where they may be stowed or operated. The 
intentions underlying the overlay designs were (1) to provide guidance to the intended (perhaps 
distant) destination for the RPCM ORU being transported and (2) to assist the operator to avoid 
confounding the needs for rotation or translation inputs from the Robotic Workstation hand 
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controllers. Most of the symbols used in the overlays were arranged for enhanced situational 
awareness however the roll arrow provided a direct recommendation for operator action. In 
heads-up display parlance, these methods are respectively referred to as “situation guidance” and 
“command guidance” (Foyle et ah, 2002). 

Subjects using the overlays were urged to use available camera views in addition to the EE 
camera view to maintain overall situational awareness. Subjects were also urged to adopt an 
internal frame of reference as they maneuvered the RPCM to the destination receptacle while 
referring to the EE view. As shown in Figure 4, much of the EE camera view was occupied by 
the grappled ORU, and overlay symbols were superimposed over the image of the RPCM. Since 
the overlays were registered with the EE during RPCM insertion, the overlays remain aligned 
with the RPCM features within the EE camera view. The (modified truncated cone) grapple 
target appeared centered in the EE camera view. In the case of the relatively small RPCM ORU, 
some of the background beyond the grappled ORU was in focus at the lateral margins of the EE 
camera view. 



RPCM 
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Figure 4. Overlay Symbols During Initial Stages of Alignment 

Figure 4 illustrates the overlay symbols likely to be encountered during the early stages of an 
ORU maneuver. The camera aiming point is marked by the intersection of the vertical and 
horizontal magenta lines at the center of the EE camera view. This point coincides with the 
center of the truncated cone target (diagonal cross) used to grapple the RPCM. The magenta 
lines remain superimposed on the field of view at all times, but they are “overwritten” by any 
other coincident overlay symbols. 
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Roll (rotation about the X-axis) 

If alignment of the RPCM and the destination receptacle requires rotation of the RPCM about the 
X-axis, a curved roll recommendation arrow appears, originating on the +Y axis (magenta center 
line) and pointing in the direction to correct the roll alignment error (Figure 4). The length of the 
arrow indicates the magnitude of the error. A left roll recommendation is indicated by an arrow 
curving to the left above the +Y axis, while a right roll recommendation (Figure 4) is indicated 
by an arrow curving to the left below the +Y axis. When the roll alignment error has been 
minimized within the allowable tolerance, the roll arrow vanishes altogether. It should be 
observed that roll motions are about the centerline of the grapple fixture, which is shown at the 
lower edge (extreme +Z direction) of the EE monitor display - not about the camera aiming 
point. The offset between the grapple fixture and the grapple target leads to a preferred order of 
operations in aligning the RPCM with the destination receptacle, as described below. 

Rotation about the Y-axis and Z-axis 

Extending outward from the central camera aiming point is a green “rotation rubber band” line, 
which tenninates at the center of the circular camera position marker. If the camera position 
marker is not within the field of view of the EE camera, the rubber band extends in the direction 
of the camera position marker to the edge of the monitor display field. 

The camera position marker (Figure 4) projects the location of a point in the Y-Z plane just 
outside the destination receptacle for the RPCM. The center of the circle marks the point in this 
Y-Z plane that lies in the +X direction from the camera lens. If the camera’s position marker 
coincides with its aiming point, the axis of the camera (and hence the longitudinal axis of the 
RPCM) is parallel to the X-axis of the receptacle. The position of the camera aiming point 
relative to the camera position marker depends only on pitch and yaw inputs from the rotation 
(right) hand controller “stick”. Pitching forward with the hand controller moves the camera 
aiming point down relative to the camera position marker in the monitor view ( i.e ., in the +Z 
direction). Yawing to the right with the hand controller moves the camera aiming point to the 
right relative to the camera position marker (i.e., in the +Y direction). The rotation rubber band 
thus shortens or lengthens in accord with the pitch/yaw error of the camera. Translation inputs 
from the left (knob) hand controller do not affect the position of the camera aiming point relative 
to the camera position marker. 

Translation in the Y or Z direction 

A second green line, the translation rubber band, extends outward from the camera position 
marker to the center of the target location marker. The target location marker is a relatively large 
white circle, whose center is marked by a black cross (Figure 4). Optimum alignment of the 
RPCM for insertion into the receptacle at the target location occurs when the camera position 
marker and camera aiming point both coincide with the target location marker. The position of 
the camera position marker relative to the target location marker is governed by translation hand 
controller (knob) commands using the left hand. As the translation error between the camera and 
target positions decreases, the translation rubber band shortens accordingly. The position of the 
camera aiming point relative to the target location marker depends not only on pitch and yaw 
inputs from the rotation (right) hand controller “stick”, but also on inputs from the translation 
(left) hand controller “knob 
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A proximity warning is provided to advise the operator to enable the Force/Moment 
Accommodation (FMA) system prior to maneuvering the RPCM closer to the receptacle in the 
+X direction. FMA protects both the RPCM and the receptacle hardware from inadvertent 
collision damage. The warning symbol consists of a red triangle at the center of the EE monitor 
display. Figure 5 shows the nominal indications just prior to enabling FMA and beginning 
insertion with a +X translation command. 



Figure 5. Overlay Appearance with ORU in Position for Insertion 

Symbol features 

Colors for symbol elements were not selected according to any coding requirements for orbital 
robotic operations. The rubber band lines and the center of the camera position marker were 
made green to avoid any confusion with yellow cautionary or red warning symbols in the 
Robotic Workstation display repertoire. Both the camera position marker and the white target 
marker symbol were outlined in black to afford high contrast against light backgrounds. Since 
the roll recommendation arrow by itself is indicative of one component of alignment error, it was 
made yellow to advise caution. The triangle of the proximity warning indicator was made red, 
because it is indicative of a potential collision hazard and warns the operator to engage the FMA 
system. 

Elements of the symbols were projected at different “levels” in the display in order to avoid 
obscuring critical features in one symbol with features of another. At the lowest level was the 
white circle of the target position marker. The next level contained the magenta lines marking 
the center of the camera view. Above this level came the camera position marker, followed by 
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the target position marker’s central black cross. The rubber band lines and the roll 
recommendation arrow occupied the uppermost level. 

Optimized Procedure Using Overlays 

The design of the overlay system suggests an optimized procedure for pre-insertion ORU 
maneuvering, which may simplify task completion by minimizing uncertainty in selecting 
whether rotation or translation hand controller inputs are appropriate to reduce alignment error. 
This feature is most advantageous when the ORU is nearly correctly aligned with the receptacle, 
since the need for small rotation or translation corrections may be difficult to discern from cues 
in the non-augmented camera views. Since the ORU rotates and translates relative to the point 
where the EE rotation axis intersects the far (connector) surface of the ORU, and the target and 
camera aiming point are offset from this point, there is asymmetry in the interactions of the 
control commands. If the camera aiming point is moved to coincide with the target location 
marker without first minimizing the residual rotation error, subsequent rotations will introduce 
additional translation error. The resulting uncertainty may lead to time-consuming trial and error 
experimental control inputs. The following procedure was demonstrated for all the subjects 
during their overlay familiarization sessions. 

Step 1. Prior to maneuvering the ORU using the Robotic Workstation hand controls, the 
operator was urged to make use of the available camera views to ensure that no obstacles lay in 
the intended path of motion for the DMT or the grappled ORU. It was suggested that if possible, 
an overview of the starting and intended destination positions of ORU should be maintained on 
the Robotic Workstation monitors. 

Step 2. In order to prevent possible disorientation and to promote efficient mapping of the hand 
controller motions to the ORU environment, it was recommended that the error in roll orientation 
be minimized by commanding a compensating roll with the rotation hand controller in 
accordance with the roll recommendation arrow early in the maneuvering process. Once the roll 
recommendation arrow disappeared, indicating that the error was minimal, it was suggested that 
the perfonnance of other maneuvers should be more easily coordinated. 

Step 3. The next stage in the optimized alignment process involved primarily the use of the 
rotation (right) hand controller - pitching and yawing the camera along the path of the rotation 
rubber band in the direction of the camera position. Once the EE camera aiming point coincides 
with the camera position marker, the EE/ORU X-axis parallels the X-axis of the destination 
receptacle, and rotation errors are minimal. 

Step 4. Once the rotation error is minimized, the EE camera aiming point may be translated in 
the direction of the translation rubber band toward the target location marker. Translation 
commands from the left hand controller should not disturb the rotational alignment of the EE 
relative to the camera position marker. When the camera aiming point coincides with both the 
camera position marker and the target location marker, the ORU should be nearly oriented for 
insertion in the receptacle using +X translation hand controller inputs. 

Some more experienced Robotic Workstation operators felt confident in making simultaneous 
rotation and translation corrections (especially early) during maneuvering. The subjects 


10 



NASA Johnsons Space Center 


preferring to use coordinated simultaneous hand controller inputs were shown that overlays did 
not interfere with this method, but it was explained that the operator should give rotation 
corrections higher priority because of the parallax between the camera and the EE rotation axis. 

It was also explained that error minimizations giving translation corrections priority over rotation 
corrections may be used successfully in a converging iterative process to achieve the insertion 
position. This is less efficient than giving higher priority to rotation operations, which requires 
only a single iteration. 

The same correction prioritization (roll, rotation, then translation) is advantageous for alignment 
of the ORU without overlays. Initial roll correction reduces the operator’s cognitive load by 
reducing interaction between Y-axis and Z-axis rotations or translations. Large initial alignment 
errors may be reduced more quickly by applying rotational corrections about the Y-axis and Z- 
axis before translating in the Y or Z directions. 

EXPERIMENTAL PROCEDURE 

Preliminary Equipment Setup 

The operation of the DMT, all video cameras, the flood lamp, and Robotic Workstation 
equipment in the cupola was verified prior to the subject’s arrival. In addition, the overlay 
registration relative to the EE camera view was checked with the RPCM in the pre-insertion 
position. This was accomplished by the DMT console operator, who first maneuvered the 
RPCM to the anticipated pre-insertion position using the overlays as a guide. The operator then 
enabled the FMA feature of the DMT software and attempted to insert the RPCM into the 
receptacle using a pure +X translation only. The RPCM was fully inserted and adjusted to the 
most centered, neutral position. No adjustment was needed if the overlay graphics were properly 
aligned. The RPCM was then withdrawn from the receptacle by means of a pure -X translation 
to the pre-insertion position. The graphics alignment was again confirmed at this point, making 
allowances for any small offsets introduced by FMA operation. If there were any significant 
discrepancies between the RPCM and the AR overlays at the pre-insertion position, the AR 
graphics position, orientation, and roll reference values in the overlay generation software were 
corrected accordingly. If adjustments were made, the procedure outlined above was repeated. 

Preliminary Subject Activities 

Each subject was provided with a printed layman’s description of the experiment, and the main 
features included in this summary were reviewed by the experiment administrator. Any 
technical or procedural questions were answered to the subject’s satisfaction, and the subject was 
asked to read and approve the informed consent. If the subject was not familiar with the 
operation of the DMT, the console operator, a certified robotics operator, provided a brief 
introductory “walk-through” of the system, explaining any major differences between the DMT 
operation and the system(s) familiar to the subject. Following this introduction, the subject was 
conducted to the cupola for further familiarization and the actual trials. 

Overlay Familiarization 

At the beginning of the experiment session, the subject was introduced to the best use of the AR 
overlays. The experiment administrator explained the functions of the overlay features using a 
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demonstration ORU starting position. The hand controllers were placed in “vernier” mode, and 
the subject operated the DMT using the Robotic Workstation and the overlays to move the 
RPCM to the pre-insertion position. Once the subject had attempted an insertion using the 
overlays, she/he was presented with a different practice starting position and allowed to attempt a 
self-paced alignment/insertion. Throughout the demonstration and practice trials, the 
administrator offered advice and answered any questions the subject posed concerning the use of 
the overlays. Following the practice trial, the subject was asked whether he/she felt competent to 
use the overlays to perform a series of trials. If the subject felt the need for additional practice, 
more practice was provided. If the subject expressed confidence in her/his ability to use the 
overlays, the test conductor requested that the ORU be positioned for the first trial. During the 
ORU setup time for the first trial, the subject was reminded that the eight trials would be self- 
paced, and that he/she could take breaks as required. The subject was also informed that the 
trials would consist of two groups of four having the same treatment (overlays or no overlays), 
and the order of the treatments was revealed. The subject was urged to use the available camera 
views to maintain overall situational awareness during all the trials. The subject was also 
reminded of the non-competitive nature of the trials and analysis methods, with encouragement 
to perfonn the task as quickly and accurately as possible. No emphasis was placed on limiting 
rates of application or on the coordination of multiple control degrees of freedom. The subject 
was urged to adjust her/his chair for the most comfort in operating the Robotic Workstation 
controls. When the console operator had placed the ORU in the starting position for the trial, 
data recording was initiated, and control was transferred from the DMT console to the subject’s 
Robotic Workstation. The subject was then ready to begin Trial 1. The DMT’s FMA feature 
was made inactive for the initial portion of each trial. 

Experimental Trial Activities 

The first action performed by the subject during a trial with or without overlays was typically to 
locate the ORU in at least one auxiliary camera view. The subject then proceeded to maneuver 
the ORU into the pre-insertion position outside the receptacle. When the pre-insertion position 
had been attained, the subject released the Robotic Workstation controls and requested that the 
console operator enable the Force/Moment Accommodation (FMA) system in preparation for 
insertion. After receiving assurances that FMA was active, the subject attempted to insert the 
ORU into the receptacle using +X commands from the translation hand controller. Once the 
ORU had stopped moving in response to the +X command, the subject released the controls, and 
the test conductor requested that the console operator withdraw the ORU and move it to the 
starting position for the next trial. 

DATA ANALYSIS 

Objective Data Recording 

Data for the experiment were extracted from the multitude of records accumulated by the DMT 
system during the trials. The sampling interval for recording was set to 100 milliseconds. The 
records used tracked position and orientation relative to the center of rotation of the ORU, the 
Point-Of-Reference (POR). This point resides along the axis of the RMCT at the surface 
opposite the grapple fixture on the ORU. DMC terms for the roll, YZ position, and pitch/yaw 
variables are identified with the “ PORact _ ” prefix. Reference values for roll, pitch, yaw, Y 
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position and Z position were determined from values recorded during the preliminary overlay 
alignment verification. 

Roll Alignment Error 

This metric is calculated as the difference (degrees) between the pre-insertion and reference 
values for the DMT PORactr variable: 

Error „ = PORactr — PORactr , 

roll pre-insertion reference 

Pitch/yaw Alignment Error 

This metric is the angle (degrees) between the receptacle (+X) axis and the RPCM axis through 
its point of reference. If 0 pre = PORactp pre _ insertion , 9 ref = PORactp reference , 

<t> pre = PORactw pre _ insertion , and <j> ref = PORactw reference , the pitch/yaw error is 


Err ° r puchlya w 

= cos"'(cos G pre sm<f> pre cos0 ref sm</> ref + cos0 pre cos </> pre cos0 ref cos <f> ref + sin^ sin6» re/ ) 

YZ Alignment Error 

This measure is the two-dimensional difference (inches) between the pre-insertion and reference 
positions in the YZ plane. The DMT variables PORacty and PORactz record these positions for 
the reference and trial samples. 

12 2 
Error YZ = (PORacty pre _ insertion - PORacty nferenc f) + (PORactz Dre _ imertion - PORactz reference ) 

Time to Completion 

Time to completion for a trial includes the interval (seconds) during which the subject exerts 
control over the DMT and cameras prior to insertion. The interval begins when the subject 
initially adjusts the camera controls or moves the hand controllers, whichever occurs first. The 
interval ends when the subject releases the hand controllers prior to requesting that FMA be 
enabled for insertion of the RPCM. 
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RESULTS 

Perfonnance on the alignment tasks described above was examined using analysis of variance 
and post-hoc mean tests. In all but the XY alignment error comparisons, however, the variances 
between the no-overlay-then-overlay (NO-OV) order and overlay-then-no-overlay (OV-NO) 
order subject groups were significantly different, as determined by the test for homogeneity of 
variance between related samples (Bruning & Kintz, pp. 109-110). For this reason, comparisons 
between non-overlay and overlay trial means for the other performance metrics used contrast test 
techniques, which are not sensitive to non-homogeneous variances (Maxwell & Delaney, pp. 
144-150). 
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Roll alignment error 

Table 2 shows the analysis of variance for roll alignment error. Variances of the four- trial mean 
errors were significantly different for the two order groupings of subjects (p < 0.012), so a 
comparison of means for the treatments was accomplished using a contrast test. This test 
indicated that the mean roll alignment error using the overlays was significantly (p < 0.02) lower 
than the mean error without overlays, as shown in Figure 7. 


Source 

SS 

df 

MS 

F 

P 

Total 

3.38131606 

23 




Between S 

0.84484965 

11 




Order 

7.31 51 E-06 

1 

7.3151 E-06 

8.6585E-05 

0.9928 

Error: S w/in Order 

0.84484234 

10 

0.08448423 



Within S 

2.53646641 

12 




Treatment (Trt) 

1.56608732 

1 

1.56608732 

16.1402 

0.0024* 

Trt x Order Interaction 

7.4378E-05 

1 

7.4378E-05 

7.6654E-04 

0.9785 

Error: Trt x S(Order) 

0.97030471 

10 

0.09703047 




*significant at a = 0.05 level 


Table 2. Analysis of Variance for Roll Alignment Error 



Figure 7. Means Comparison for Roll Alignment Error 
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Pitch/yaw Alignment Error 

The analysis of variance for pitch/yaw alignment error is shown in Table 3. Variances of the 
four-trial mean errors were significantly (p < 0.014) different for the two order groupings of 
subjects, so a comparison of means for the treatments was accomplished using a contrast test. 
The contrast test revealed that the mean pitch/yaw alignment error using the overlays was 
significantly (p < 0.0103) lower than the mean error without overlays, as illustrated in Figure 8 
The apparent interaction between trial order and treatment was not significant at the a = 0.05 
level. 


Source 

SS 

df 

MS 

F 

P 

Total 

6.14366655 

23 




Between S 

0.8795165 

11 




Order 

0.01903151 

1 

0.01903151 

0.2212 

0.6482 

Error: S w/in Order 

0.86048499 

10 

0.0860485 



Within S 

5.26415005 

12 




Treatment (Trt) 

4.20526529 

1 

4.20526529 

50.505 

< 0.0001* 

Trt x Order Interaction 

0.22624114 

1 

0.22624114 

2.7171 

0.1303 

Error: Trt x S(Order) 

0.83264362 

10 

0.08326436 




*significant at a = 0.05 level 


Table 3. Analysis of Variance for Pitch/yaw Alignment Error 



Figure 8. Means Comparison for Pitch/yaw Alignment Error 
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YZ Alignment Error 

Results of analysis of variance for mean YZ alignment errors are shown in Table 1. Variances of 
the four-trial mean errors were not significantly different for the two order groupings of subjects 
(p > 0.98), so a comparison of means for the treatments was accomplished using a standard t-test. 
The mean YZ alignment error using the overlays was significantly (p < 0.01 1) lower than the 
mean error without overlays, as shown in Figure 6. 


Source 

SS 

df 

MS 

F 

P 

Total 

0.5661955 

23 




Between S 

0.12289323 

11 




Order 

0.00039767 

1 

0.00039767 

0.0325 

0.8606 

Error: S w/in Order 

0.12249556 

10 

0.012249556 



Within S 

0.44402631 

12 




Treatment (Trt) 

0.28938562 

1 

0.28938562 

18.8386 

0.0015* 

Trt x Order Interaction 

0.00102759 

1 

0.00102759 

0.0669 

0.8012 

Error: Trt x S(Order) 

0.1536131 

10 

0.01536131 




*significant at a = 0.05 level 

Table 1. Analysis of variance for YZ alignment error 



Figure 6. Means Comparison for YZ Alignment Error 
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Time to Completion 

As shown in Table 4, the analysis of variance for time to completion revealed significant 
interaction between the treatment and order variables (p < 0.05). Variances of the four-trial 
mean times to completion for the two order groupings of subjects were also non-homogeneous at 
the p < 0.0001 level, so a comparison of means for the four treatment-by-order combinations was 
performed using contrast tests. The results of these tests are shown in Table 5. Whereas the 
difference in time to completion was significantly less (p < 0.0359) with overlays than without 
overlays in the NO-OV trials, the similar difference for the times to completion in the OV-NO 
trials was not significant (p > 0.16). Some insight into this outcome is provided by examining 
times to complete individual NO-OV and OV-NO trials in Figure 10 and Figure 11, respectively. 
The vertical scales in these figures are the same, and the apparent difference between the plots 
seems to be associated with the performance of subjects S05, SI 1, and S13 during the first four 
(NO) trials. S05 required much more time to complete the initial NO trial than any others. 
Completion times for S 13 decreased monotonically through the NO trials, while SI 1 exhibited 
more variability, with the third trial requiring more time. 


Source 

SS 

df 

MS 

F 

P 

Total 

450197.634 

23 




Between S 

200367.955 

11 




Order 

6133.28468 

1 

6133.28468 

0.3158 

0.5865 

Error: S w/in Order 

194234.67 

10 

19423.467 



Within S 

249829.679 

12 




Treatment (Trt) 

137814.721 

1 

137814.721 

19.1227 

0.0014* 

Trt x Order Interaction 

39946.3826 

1 

39946.3826 

5.5428 

0.0403* 

Error: Trt x S(Order) 

72068.5763 

10 

7206.85763 




‘significant at a = 0.05 level 

Table 4. Analysis of Variance for Time to Completion 


Conditions 

Numerator 

Denominator 

F 

df 

P 

NO-OV, NO 

NO-OV, OV 

163077.584 

20102.7009 

8.1122 

5.8616 

0.0359* 

NO-OV, NO 

OV-NO, NO 

38692.3905 

20897.1915 

1.8516 

6.2761 

0.2225 

NO-OV, NO 

OV-NO, OV 

101047.305 

22623.6453 

4.4664 

7.1249 

0.0724 

NO-OV, OV 

OV-NO, NO 

42900.7002 

4006.6794 

10.7072 

9.6051 

0.0096* 

NO-OV, OV 

OV-NO, OV 

7387.27676 

5733.13312 

1.2885 

8.3692 

0.2892 

OV-NO, NO 

OV-NO, OV 

14683.5196 

6527.6238 

2.2494 

9.3361 

0.1679 


‘significant at a = 0.05 level 


Table 5. Means Comparison Tests for Time to Completion 
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Figure 9. Means Comparison for Time to Completion 




-SOI 

■ 

S03 


S05 
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-S11 
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Figure 10. Elapsed Times to Completion for NO-OV Trials 


19 


NASA Johnsons Space Center 



— S02 
— ■— S04 
S06 
S08 
— * — S10 
— • — S12 


Figure 11. Elapsed Times to Completion for OV-NO Trials 


Post-trials Survey 

The results of the survey administered following the subject’s completion of all eight trial is 
presented in Table 6. In general, it is apparent that the subjects believed the overlays were 
helpful. 


Statement 

1) Instructions concerning the use of the overlays were clear. 

2) Adequate practice with the overlays was provided prior to the 

trials. 

3) Functions of the overlays were easy to understand. 

4) Arrangement of the overlays seemed intuitive. 

5) Overlays were easy to distinguish from their background. 

6) Rubber band features of the overlays clearly indicated required 

hand controller inputs. 

7) Overlays helped to compensate for adverse lighting conditions. 

8) Overlay information was helpful/useful in performing the ORU 

alignment operation. 

9) Overlay information did not hinder the completion of the ORU 

alignment. 

10) The overlays increased confidence in the results of the 

alignment task. 

11) There were no noticeable differences in task difficulty with or 

without overlays. 

12) The overlays allowed the speed of alignment to be increased. 

13) The use of overlays interfered with overall situation awareness. 


Completely 

Agree Agree Neutral 

10 2 0 


Completely 
Disagree Disagree 

0 0 


10 2 0 

4 8 0 

4 7 1 

8 3 1 


0 

0 

0 

0 


0 

0 

0 

0 


4 6 0 2 

9 12 0 


0 

0 


12 


0 0 0 


0 


12 


0 0 0 


0 


10 2 0 


0 


0 


0 

10 

0 


0 1 
1 0 
1 1 


6 

1 

1 


4 

0 

9 


Table 6. Questionnaire Responses 
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DISCUSSION 

The results of this experiment indicate that overlays improve performance in maneuvering an 
ORU in preparation for inserting it into a receptacle. Despite the small number of subjects 
available to participate, three of the four performance metrics showed statistically significant 
improvements in pre-positioning accuracy using overlays. Trial completion time results are 
slightly less clear cut, but variability in performance was reduced for all metrics during trials 
using overlays. 

In the cases of all metrics except roll error, there may be evidence of increased performance with 
practice (Figures 6, 8, and 9). For trials without overlays, the subjects experiencing no overlays 
in the first four trials showed greater error than did the subjects experiencing no overlays in the 
last four trials. Similarly, for trials with overlays, the subjects experiencing overlays in the first 
four trials showed greater error than did the subjects having overlays in the last four trials. Only 
in the case of pitch/yaw error, however, is the order/treatment interaction significant at the a = 
0.05 level. 

Position Determination Assumptions 

A fundamental assumption underlying the use of overlays to improve robotic operations is that 
the position of the end effector is accurately known. For this investigation, POR location was 
taken directly from the calculations by the monitoring program running on the DMT console 
computer. The location was calculated by using known component dimensions and joint angle 
measurements. The uncertainty in the POR location accumulates with the degrees of freedom 
available for robotic motions. The DMT location data was checked prior to each trial and proved 
adequately stable for the purpose of the experiment. In actual operations, however, additional 
sources for error will likely be encountered in calculating payload position from joint angles and 
robot arm dimensions. A method of directly estimating the payload position relative to the 
destination receptacle independent of the robot geometry is essential for practical applications 
using overlays. This may be provided by an independent position sensing system or, if camera 
views of the scene are available, through photogrammetric methods. Investigations into the 
latter are currently underway at JSC in collaboration with Wayne State University. 

Subjects 

Experience using the Robotic Workstation varied widely among the subjects, and differences in 
performance were not directly attributable to this factor. If possible, future studies relating to the 
effectiveness of overlays should balance subject groups according to prior robotic experience. 
Such studies should be planned to include more subjects. This implies that future work should 
consider using computer-based simulations or robotic systems other than the DMT, which 
necessitates specialized training and certification for operators. A larger pool of subjects having 
a wide range of experience might be used to investigate potential AR applications, such as the 
use of overlays as a training aid. 

Symbol Rationale 

The symbol set chosen is intended to provide guidance in cases of extreme misalignment. It is 
also designed to reduce the likelihood of an operator’s confounding rotation and translation cues 
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as the ORU approaches pre-insertion alignment. The inclusion of the camera position marker 
allows unambiguous, decoupled tracking of rotation and translation by the ORU. 

With the exception of the roll recommendation arrow, all symbols afforded the operator situation 
guidance. Only the roll recommendation arrow provided command guidance, demanding that 
the operator respond with a particular control input. Since the ORU maneuvering task lends 
itself to an internal frame of reference, it is conceivable that the roll recommendation arrow 
might be replaced by a situation guidance symbol, similar to the artificial horizon commonly 
used in aircraft attitude indicators. Comments by several of the subjects in this study suggested 
that arrows along the rubber bands might provide helpful prompting for the correct direction of 
operation for the hand controls. Such a modification of the rubber band lines would essentially 
convert their mode of operation from offering situation guidance to providing coimnand 
guidance. The relative merits of the command guidance scheme compared to that of situation 
guidance in the ORU positioning task affords potential opportunities for future research. 

Conclusions and Future Work 

This study illustrates some of the potential for the application of AR overlays in robotic control 
systems. If symbolic AR overlays were included in the operator interface for future robotic 
operations, such systems might accrue benefits in addition to reduced errors and task time. With 
appropriate simulation software, the overlays might provide an advantage in training for robotic 
applications. A simulation having the appropriate fidelity and AR overlays to prompt the correct 
actions might provide valuable “refresher” training between infrequent actual robotic tasks 
during extended exploration missions. This hypothesis could be tested within a subject pool on 
the ground by means of test-retest performance evaluations over varying lengths of time without 
intervening practice. 

The relative merits of overlays providing situational guidance versus their providing command 
guidance could be studied by means of simulations of moderate fidelity. Any such research 
should also address the effects of mixed guidance modes on operator performance. 

Because the graphics are relatively simple and the computational requirements are low the 
system can even be implemented on existing hardware (Avionics Vision Unit) used on shuttle 
and space station. A previous technology development project (Maida, et al) demonstrated this 
feasibility. The potential benefits provided by augmented reality techniques in robotic control 
should be considered in the modification of existing systems or the designing of future systems 
for operations in space. 
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